Image generation system, program and method, and simulation system, program and method

ABSTRACT

This system of the present invention uses computer graphics techniques to generate a virtual sensor image. The computer graphics include: a means for creating a scenario of an object present in the image; a means for performing modeling for each object in the computer graphics on the basis of a scenario; a means for performing shading for each model of the modeling result; a means for outputting only one component of a shaded image; and a means for generating a depth image on the basis of three-dimensional profile information for each object in the computer graphics.

CROSS-REFERENCES TO RELATED APPLICATIONS

This Application claims the benefit of priority and is a Continuationapplication of the prior International Patent Application No.PCT/JP2017/033729, with an international filing date of Sep. 19, 2017,which designated the United States, and is related to the JapanesePatent Application No. 2016-197999, filed Oct. 6, 2016 and JapanesePatent Application No. 2017-092950, filed May 9, 2017, the entiredisclosures of all applications are expressly incorporated by referencein their entirety herein.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a simulation system, a simulationprogram and a simulation method of a recognition function module for animage varying with position shifting information of a vehicle by the useof a virtual image of a near infrared sensor and a laser beam sensor ofa LiDAR.

2. Description of Related Art

At the present time, for the purpose of realizing automatic driving ofvehicles such as ADAS (advanced driver assistance system) or the like todetect and avoid the possibility of an accident in advance, varioustests have actively been conducted by recognizing images of a camerainstalled on a vehicle to detect objects such as other vehicles, walkersand a traffic signal in accordance with an image recognition techniqueto perform control to automatically decrease the speed of the vehicleand avoid the objects and the like. In the case of the above experimentsystem, it is particularly important to synchronously control the entiresystem with a real-time property and a high recognition rate.

An example of an automatic driving support system is a travel controlsystem disclosed, for example, in Patent Document 1. The travel controlsystem disclosed in this Patent Document 1 is aimed at realizing anautomatic driving system with which a vehicle can travel on apredetermined traveling route by detecting road markings such as a lanemarker, a stop position and the like around own vehicle on a road anddetecting solid objects such as a plurality of mobile objects/obstacleslocated around the own vehicle to determine a traveling area on the roadwhile avoiding collision with solid objects such as a traffic signal anda signboard.

Incidentally, for the purpose of performing control by recognizing anoutside peripheral situation with onboard sensors, it is required todetermine vehicles, bicycles and walkers which belong to categories of aplurality of mobile objects and a plurality of obstacles, and detectsinformation about positions and speeds thereof. Furthermore, for drivingown vehicle, it is required to determine the meanings of paints such asa lane marker and a stop sign on a road, and the meanings of trafficsigns. As a vehicle-mounted camera for detecting outside informationaround own vehicle, it is considered effective to use an imagerecognition technique with an image sensor of a camera.

[Patent Document] Japanese Unexamined Patent Application Publication No.2016-99635

BRIEF SUMMARY OF THE INVENTION

In order to realize automatic driving of a vehicle, the vehicle itselfhas to recognize the surrounding environment. For this purpose, it isneeded to accurately measure the distance between the vehicle itself anda surrounding object. The technique for performing distance measurementhas been developed with the following devices which have been alreadyinstalled in many marketed vehicles for realizing driving assisttechniques such as lane keeping, cruise control and automatic braking.

-   -   Stereoscopic camera: The distance is calculated in accordance        with the principle of triangulation by the use of two cameras in        the same manner as human's eyes.    -   Infrared depth sensor: The distance is calculated by radiating        an infrared ray pattern, imaging reflection thereof with an        infrared ray camera, and calculating the distance with reference        to the dislocation of the pattern (phase difference).    -   Ultrasonic wave sensor: The distance is calculated on the basis        of the time taken from emission of a ultrasonic wave to        reception of the reflected wave thereof.    -   Millimeter wave radar: The distance is calculated on the basis        of the time taken from emission of a millimeter radar wave to        reception of the reflected wave thereof in the same manner as a        ultrasonic wave sensor.    -   LiDAR (Light Detection and Ranging): The distance is calculated        by the use of a laser light, in the same manner as a ultrasonic        wave sensor or a millimeter wave radar, on the basis of the time        (TOF: Time of Flight) taken from emission to reception of the        reflected wave thereof.

While there are a plurality of methods as described above, each methodhas both advantages and disadvantages. In the case of the stereoscopiccamera, while a distance can easily and accurately be measured by athree-dimensional view, two cameras have to be separated by at least 30cm resulting in the limit of miniaturization.

The infrared depth sensor and the ultrasonic wave sensor areadvantageous in low costs, but substantial attenuation is caused bydistance. Because of this, in the case where the distance to the objectis greater than several tens of meters, accurate measurement becomesdifficult, or measurement itself becomes impossible. Contrary to this,the millimeter wave radar and the LiDAR result in less attenuation evenover a long distance, so that it is possible to perform accuratemeasurement even over a long distance. While there are problems that theapparatus becomes expensive and that it is difficult to reduce the size,installation thereof on vehicles is considered to accelerate by thefuture research and development.

As has been discussed above, in order to accurately measure the distanceto the object from a short distance to a long distance, it is apractical means at the present time to selectively use differentsensors. Besides the automatic driving of vehicles, promisingapplications of the sensors include the technique of detecting themotion of a head for preventing a driver from napping in a vehicle, thetechnique of detecting gestures, and the technique of avoiding anobstacle for automatic moving robot.

Incidentally, it is regarded as indispensable for future automaticdriving to collect a large number of photographed images taken byvarious sensors as described above to improve the recognition rate ofimages by a deep learning recognition technique.

However, while it is practically impossible to collect test data byendlessly driving a vehicle in the actual world, it is an importantissue how to carry out the above verification with a sufficient realityof an actually substitutable level. For example, in the case where anoutside environment is recognized by an image recognition technique withcamera images, the recognition rate is substantially changed by externalfactors such as the weather around own vehicle (rain, fog or the like)and the time zone (night, twilight, backlight or the like) to influencethe detection result. As a result, with respect to mobile objects,obstacles and paints on a load around own vehicle, there are increasedmisdetection and undetection. Such misdetection and undetection of animage recognition means can be resolved with a deep leaning (machinelearning) technique having a highest recognition rate by increasing thenumber of samples for learning.

However, it has a limit to extract learning samples during actuallydriving on a load, and it is not realistic as a development technique tocarry out a driving test and sample collection after meeting severeweather conditions such as rain, backlight, fog or the like while suchconditions are difficult to reproduce only with a rare opportunity.

On the other hand, for the purpose of realizing fully automatic drivingin future, the above image recognition of camera images would notsuffice. This is because camera images are two-dimensional images sothat, while it is possible to extract objects such as vehicles, walkersand a traffic signal and the like by image recognition, it is impossibleto detect the distance to each picture element of the object.Accordingly, a sensor using laser beams called LiDAR and a sensor usingnear infrared rays are highly anticipated as means for dealing with theabove issues. It is therefore possible to substantially improve thesafety of a vehicle during driving by combining a plurality of differenttypes of sensors as described above.

In order to solve the problem as described above, the present inventionis related to the improvement of the recognition rate of target objectssuch as other vehicles peripheral to own vehicle, obstacles on the road,and walkers, and it is an object of the present invention to improvereality of the driving test of a vehicle and sample collection byartificially generating images which are very similar to actuallyphotographed images taken under conditions, such as severe weatherconditions, which are difficult to reproduce. In addition, it is anobject of the present invention to build a plurality of different typesof sensors in a virtual environment and generate images of each sensorby the use of a CG technique. Furthermore, it is an object to provide asimulation system, a simulation program and a simulation method forperforming synchronization control with CG images which are generated.

In order to accomplish the object as described above, the presentinvention is related to a system, a program and a method of generating,as computer graphics, a virtual image which is input to a sensor unit,comprising:

a scenario creation unit which creates a scenario relating to locationsand behaviors of objects existing in the virtual image;

a 3D modeling unit which performs modeling of each of the objects on thebasis of the scenario;

a 3D shading unit which performs shading of each model generated by themodeling unit and generates a shading image of each model;

a component extraction unit which extracts and outputs a predeterminedcomponent contained in the shading image as a component image; and

a depth image generation unit which generates a depth image in which adepth is defined on the basis of three-dimensional profile informationabout each object in the component image.

In the case of the above invention, it is preferred that the componentis an R component of an RGB image.

Also, in the case of the above invention, it is preferred to furtherprovide a gray scale conversion unit which performs gray scaleconversion of the component.

The present invention is related to a system, a program and a method ofgenerating, as computer graphics, a virtual image which is input to asensor unit, comprising:

a scenario creation unit which creates a scenario relating to locationsand behaviors of objects existing in the virtual image;

a 3D modeling unit which performs modeling of each of the objects on thebasis of the scenario;

a 3D shading unit which performs shading of each model generated by themodeling unit and generates a shading image of each model; and

a depth image generation unit which generates a depth image in which adepth is defined on the basis of three-dimensional profile informationabout each of the objects, wherein

the shading unit is provided with:

a function to perform shading only of a predetermined portion of themodel on which is reflected a light beam emitted from the sensor unit;and

a function to output only a three-dimensional profile of thepredetermined portion, and wherein

the depth image generation unit generates a depth image of each of theobjects on the basis of the three-dimensional profile of thepredetermined portion.

In the case of the above invention, it is preferred that the sensor unitis a near infrared sensor. Also, in the case of the above invention, itis preferred that the sensor unit is a LiDAR sensor which detectsreflected light of emitted laser light.

In the case of the above invention, it is preferred that the scenariocreation unit is provided with a mechanism to determinethree-dimensional profile information of objects, behavior informationof objects, material information of objects, parameter information oflight sources, positional information of cameras and positionalinformation of sensors.

In the case of the above invention, it is preferred to further provide adeep learning recognition learning unit which acquires, as teacher data,and performs training of a neural network by back propagation on thebasis of the component image, the depth image generated by the depthimage generation unit, and the teacher data.

In the case of the above invention, it is preferred to provide a deeplearning recognition learning unit which acquires, as teacher data, anirradiation image and a depth image on the basis of actual photography,and performs training of a neural network by back propagation on thebasis of the image obtained by the shading unit as a result of shading,the depth image generated by the depth image generation unit, and theteacher data.

In the case of the above invention, it is preferred to further provide

a TOF calculation unit which calculates, as TOF information, a timerequired from irradiation of a light beam to reception of a reflectedlight thereof on the basis of the depth image generated by the depthimage generation unit;

a distance image generation unit which generates a distance image on thebasis of the TOF information calculated by the TOF calculation unit; and

a comparison evaluation unit which compares the distance image generatedby the distance image generation unit and the depth image generated bythe depth image generation unit.

In the case of the above invention, it is preferred that the modelingunit has a function to acquire the result of comparison by thecomparison evaluation unit as feedback information, adjust conditions ofthe modeling on the basis of the acquired feedback information, andperform modeling again.

In the case of the above invention, it is preferred that the modelingunit repeats the modeling until matching error of the comparison resultby the comparison evaluation unit becomes smaller than a predeterminedthreshold by repeating acquisition of the feedback information on thebasis of the modeling and the comparison.

Furthermore, the present invention is related to a simulation system, aprogram and a method of a recognition function module for an imagevarying in correspondence with position shifting information of avehicle, comprising:

a positional information acquisition unit which acquires positionalinformation of the vehicle in relation to a surrounding object on thebasis of a detection result by a sensor unit;

an image generation unit which generates a simulation image forreproducing an area specified by the positional information on the basisof the positional information acquired by the positional informationacquisition unit;

an image recognition unit which recognizes and detects a particularobject by the recognition function module in the simulation imagegenerated by the image generation unit;

a positional information calculation unit which generates a controlsignal for controlling behavior of the vehicle by the use of therecognition result of the image recognition unit, and changes/modifiesthe positional information of own vehicle on the basis of the generatedcontrol signal; and

a synchronization control unit which controls synchronization among thepositional information acquisition unit, the image generation unit, theimage recognition unit and the positional information calculation unit.

In the case of the above invention, it is preferred that thesynchronization control unit further comprises:

a unit of packetizing the positional information in a particular formatand transmitting the packetized positional information;

a unit of transmitting the packetized data through a network or atransmission bus in a particular device;

a unit of receiving and depacketizing the packetized data; and

a unit of receiving the depacketized data and generating an image.

In the case of the above invention, it is preferred that thesynchronization control unit transmits and receives signals among therespective units in accordance with UDP (User Datagram Protocol).

In the case of the above invention, it is preferred that the positionalinformation of the vehicle includes information about any of XYZcoordinates of road surface absolute position coordinates of thevehicle, XYZ coordinates of road surface absolute position coordinatesof tires, Euler angles of own vehicle and a wheel rotation angle.

In the case of the above invention, it is preferred that the imagegeneration unit is provided with a unit of synthesizing athree-dimensional profile of the vehicle by computer graphics.

In the case of the above invention, it is preferred that, as the abovevehicle, a plurality of vehicles are set up for each of which therecognition function operates, that

the positional information calculation unit changes/modifies thepositional information of each of the plurality of vehicles by the useof information about the recognition result of the recognition unit, andthat

the synchronization control unit controls synchronization among thepositional information acquisition unit, the image generation unit, theimage recognition unit and the positional information calculation unitfor each of the plurality of vehicles.

In the case of the above invention, it is preferred that the imagegeneration unit is provided with a unit of generating a different imagefor each sensor unit.

Also, in the case of the above invention, it is preferred that there isprovided, as the sensor unit, with any or all of an image sensor, aLiDAR sensor, a millimeter wave sensor and an infrared sensor.

In the case of the above invention, it is preferred that the simulationsystem is provided with a unit of generating images corresponding to aplurality of sensors, a recognition unit supporting the generatedimages, a unit of performing the synchronization control by the use ofthe plurality of the recognition results.

In the case of the invention related to the simulation system, theprogram and the method, it is further preferred that the above inventionof the image generation system, the image generation program and theimage generation method are provided as the image generation unit asdescribed above, and that

the depth image generated by the depth image generation unit of theimage generation system is input to the image recognition unit as thesimulation image.

As has been discussed above, in accordance with the above inventions, itis possible for learning of a recognition function module such as deeplearning (machine learning) to increase the number of samples byartificially generating images such as CG images which are very similarto actually photographed images and improve the recognition rate byincreasing learning efficiency.

Specifically, in accordance with the present invention, it is possibleto artificially and infinitely generate images with a light source, anenvironment and the like which are do actually not exist by making useof a means for generating and as synthesizing CG images with highreality on the basis of a simulation model. Test can be conducted as towhether or not target objects can be recognized and extracted byinputting the generated images to the recognition function module in thesame manner as inputting conventional camera images, and performing thesame process with the generated images as with the camera images, andtherefore it is possible to perform learning with such types of imagesas conventionally difficult or impossible to acquire or take, andfurthermore to effectively improve the recognition rate by increasinglearning efficiency.

Furthermore, synergistic effects can be expected by simultaneously usingdifferent types of sensors such as a millimeter wave sensor and a LiDARsensor capable of extracting a three-dimensional profile of an object inaddition to an image sensor capable of acquiring a two-dimensional imageand generating images of these sensors to make it possible to conductextensive tests and perform brush-up of a recognition technique at thesame time.

Incidentally, the application of the present invention covers a widefield, such as, for automatic vehicle driving, experimental apparatuses,simulators, software modules and hardware devices related thereto (forexample, a vehicle-mounted camera, an image sensor, a laser sensor formeasuring a three-dimensional profile of the circumference of avehicle), and machine learning software such as deep learning. Also,since a synchronization control technique is combined with a CGtechnique capable of realistically reproducing actually photographedimage, the present invention can be widely applied to other fields thanthe automatic driving of a vehicle. For example, potential fields ofapplication include a simulator of surgical operation, a militarysimulator and a safety running test system for robot, drone or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the overall configuration of an imagegeneration system for generating a virtual images in accordance with afirst embodiment.

FIGS. 2A, 2B and 2C are explanatory views for showing the process ofcollecting three-dimensional data by actually driving a vehicle on aroad.

FIGS. 3A and 3B are explanatory views for showing collection of thethree-dimensional profile of a test vehicle.

FIG. 4 shows a gray scale image acquired by a near infrared sensor.

FIG. 5 shows a distance image acquired by a near infrared sensor.

FIG. 6 is a block diagram showing the overall configuration of an imagegeneration system for generating a virtual images in accordance with asecond embodiment.

FIG. 7 is an explanatory view for showing TOF of laser light.

FIGS. 8A to 8C are explanatory views for showing the configuration andoperational mechanism of a LiDAR.

FIG. 9 is an explanatory view for showing beam irradiation of a LiDAR.

FIG. 10 is an explanatory view for showing irradiation of laser beams ofa LiDAR onto target objects.

FIG. 11 is a block diagram for explaining a neural network and a backpropagation in accordance with a third embodiment.

FIG. 12 is an explanatory view for explaining a neural network.

FIG. 13 is a block diagram for explaining an image quality evaluationsystem for depth images in accordance with a fourth embodiment.

FIGS. 14A and 14B are explanatory views for explaining the concept ofTOF and the relationship between a projection light pulse and a lightreception pulse.

FIG. 15 is a block diagram for explaining a synchronization simulationsystem in accordance with a fifth embodiment.

FIG. 16 is a block diagram for explaining the structure of a clientside.

FIG. 17 is a block diagram for explaining the structure of a simulatorserver side.

FIGS. 18A and 18B are flow charts for explaining the structure relatingto UDP synchronization control, image generation and image recognition.

FIG. 19 is a flow chart for showing the operation of a synchronizationcontrol simulator.

FIG. 20 is a block diagram for explaining a plurality of synchronizationsimulation systems in accordance with a sixth embodiment.

FIG. 21 is a block diagram for explaining a plurality of UDPsynchronization control systems.

FIG. 22 is a block diagram for explaining a plurality of deep learningrecognition units in accordance with a seventh embodiment.

FIG. 23 is a block diagram for explaining a plurality of deep learningrecognition units provided with a material imaging means.

DETAILED DESCRIPTION OF THE INVENTION First Embodiment (OverallConfiguration of a Near Infrared Ray Virtual Image Generation System)

In what follows, with reference to the accompanying drawings, a nearinfrared ray virtual image generation system in accordance with thepresent invention will be explained in detail. In the case of thepresent embodiment, for the purpose of replacing photographed imagestaken by various types of sensors which are regarded indispensable forautomatic driving, a system is built which generates images, whichconsiderably resemble photographed images, by a CG technique. FIG. 1 isa block diagram for generating near infrared virtual images.

Incidentally, the near infrared ray virtual image generation system inaccordance with the present embodiment is implemented, for example, byexecuting software installed in a computer to build virtual variousmodules on an arithmetic processing unit such as a CPU installed in thecomputer. Meanwhile, in the context of this document, the term “module”is intended to encompass any function unit capable of performingnecessary operation, as implemented with hardware such as a device or anapparatus, software capable of performing the functionality of thehardware, or any combination thereof.

As shown in FIG. 1, the near infrared ray virtual image generationsystem in accordance with the present embodiment is provided with ascenario creation unit 10, a 3D modeling unit 11, a 3D shading unit 12,an R image gray scale conversion unit 13 and a depth image generationunit 14.

The scenario creation unit 10 is a means for creating a scenario datawhich determines what CG is to be generated. This scenario creation unit10 is provided with a means for determining three-dimensional profileinformation of target objects, behavior information of target objects,material information of target objects, parameter information of lightsources, positional information of cameras and positional information ofsensors. For example, in the case of CG for use in automatic driving,while there are a number of target objects such as a road, a building, avehicle, a walker, a bicycle, a road side strip and a traffic signal ina virtual space, scenario data defines what target objects exist in whatpositions (coordinates, altitudes) of the virtual space and what motionis taken in what direction, and also defines the position (view point)of a virtual camera in the virtual space, the number and types of lightsources, the positions and direction of each light source, movement andbehavior of the target objects in the virtual space and the like.

It is determined first by this scenario creation unit 10 what kinds ofCG images are generated. The 3D modeling unit 11 generates 3D images inaccordance with the scenario created by the scenario creation unit 10.

The 3D modeling unit 11 is a module for generating the profile of anobject in the virtual space by setting the coordinates of each vertexfor forming the exterior shape of the object and the profile of theinternal structure thereof and setting the parameters of equationsrepresenting the boundaries and surfaces of the profile to build thethree-dimensional shape of the object. Specifically, this 3D modelingunit 11 performs modeling of information such as the 3D profile of aroad, the 3D profile of a vehicle traveling on the road and the 3Dprofile of a walker.

The 3D shading unit 12 is a module for generating actual 3D CG by theuse of each 3D model data D101 generated by the 3D modeling unit 11 torepresent shading of an object of 3D CG by a shading process so that astereoscopic real image is generated in accordance with the position ofa light source and the intensity of light.

The R image gray scale conversion unit 13 is a module for functioning asa component extraction unit which extracts predetermined componentscontained in a shading image transmitted from the 3D shading unit 12,and as a gray scale conversion unit which converts the extractedcomponent image to a gray scale image. Specifically, the R image grayscale conversion unit 13 extracts, as a component image, the R componentfrom the shading image D103 which is an RGB image transmitted from the3D shading unit 12, converts the R component of the extracted Rcomponent image to a gray scale image, and outputs a gray scale imageD104 (Img(x, y), x: horizontal coordinate value, and y: verticalcoordinate value) as illustrated in FIG. 4. By this process, only the R(red) component is extracted from the shading image D103 to generate animage which is extremely close to an infrared light image. FIG. 4 showsa black/white images which are generated by converting a photographedimage of a room taken by a near infrared sensor to a gray scale image.

The depth image generation unit 14 is a module for acquiring 3D profiledata of each target object in a screen on the basis of modelinginformation D102 of each individual 3D profile model input from the 3Dshading unit 12, and generating a depth image (also called a Depth-map)105 on the basis of the distance to each target object. FIG. 5 shows animage generated by coloring the above depth image in accordance withdistance. The nearer the target object is located, the greater the redcomponent becomes, and the remoter the target object is located, thegreater the blue component becomes. The target objects located inintermediate positions are colored from yellow to green, and thereforedepth information can be obtained for all the target objects.

(Operation of the Near Infrared Ray Virtual Image Generation System)

The near infrared ray virtual image generation method of the presentinvention can be implemented by operating the near infrared ray virtualimage generation system having the structure as described above.

First, the scenario creation unit 10 creates a scenario what CG is to begenerated. For example, in the case of CG for automatic driving, thescenario creation unit 10 creates a scenario which defines in whatpositions are located a number of target object such as a road, abuilding, a vehicle, a walker, a bicycle, a road side strip and atraffic signal, and what motion is taken in what direction, and alsodefines the position of a camera, the number and types of light sources.

This scenario creation unit 10 determines what CG is to be generated.Next, modeling of information such as the 3D profile of a road, the 3Dprofile of a vehicle traveling on the road and the 3D profile of awalker is performed in accordance with the scenario created by thescenario creation unit 10. Incidentally, modeling means can easily beimplemented by, for example with respect to roads, using “high precisionmap database” which is made by moving a number of vehicles each of whichis equipped with a vehicle-mounted device 1 b as illustrated in FIG. 2A,making a 3D map from data collected as illustrated in FIG. 2B, andlinking the elements of each road by the use of a vectorized drawing asillustrated in FIG. 2C.

Next, the 3D modeling unit 11 acquires or generates a 3D profile modelof each target object as required on the basis of scenario informationD100 created by the scenario creation unit 10. Then, the 3D shading unit12 generates actual 3D CG by the use of each 3D model data D101 which isgenerated by the 3D modeling unit 11.

Also, the R component shading image D103 transmitted from the 3D shadingunit 12 is converted to a gray scale image of the R image as illustratedin FIG. 4, and output as a gray scale image D104 (Img(x, y), x:horizontal coordinate value, and y: vertical coordinate value). On theother hand, the 3D shading unit 12 generates the modeling informationD102 of each individual 3D profile model from which is obtained 3Dprofile data of each target object in a screen, and the depth imagegeneration unit 14 generates a depth image D105 (a (x, y), x: horizontalcoordinate value, and y: vertical coordinate value) on the basis of thedata.

Then, after gray scale conversion by the process as described above,image recognition is performed by the use of the gray scale image D104and the depth image D105 which are transmitted as output images of thepresent embodiment.

Second Embodiment

In what follows, with reference to the accompanying drawings, a secondembodiment of the system in accordance with the present invention willbe explained in detail. Meanwhile, in the description of the presentembodiment, like reference numbers indicate functionally similarelements as the above first embodiment unless otherwise specified, andtherefore no redundant description is repeated.

(Overall Configuration of a LiDAR Sensor Virtual Image GenerationSystem)

In the case of the present embodiment, a system making use of a LiDARsensor will be described. The system in accordance with the presentembodiment is implemented as illustrated in FIG. 6 and includes ascenario creation unit 10, a 3D modeling unit 11, a shading unit 15 anda depth image generation unit 16.

The shading unit 15 of the present embodiment is a module for generatingactual 3D CG by the use of each 3D model data D101 generated by the 3Dmodeling unit 11 to represent shading of an object of 3D CG by a shadingprocess so that a stereoscopic real image is generated with the positionof a light source and the intensity of light. Particularly, the shadingunit 15 of the present embodiment is provided with a laser irradiatedportion extraction unit 15 a which extracts a 3D profile only from aportion which is irradiated with laser light, performs shading of theextracted 3D profile and outputs a shading image D106. Also, since thereflected light of laser light has no color component such as RGB, theshading image D106 is output from the shading unit 15 directly as a grayscale image.

Also, the depth image generation unit 16 is a module for acquiring the3D profile data of each target object in a screen on the basis ofmodeling information D102 of each individual 3D profile model input fromthe 3D shading unit 12, and generating a depth image (also called aDepth-map) 105 on the basis of the distance to each target object.Particularly, the depth image generation unit 16 of the presentembodiment outputs a depth image D108 extracted only from a portionwhich is irradiated with laser light by the laser irradiated portionextraction unit 16 a.

(Operation of the LiDAR Sensor Virtual Image Generation System)

Next, the operation of the LiDAR sensor virtual image generation systemhaving the structure as described above will be explained.

In the case of near infrared light, the image shown in FIG. 5 can becaptured at the same time from among the respective target objectsprocessed by 3D modeling. Contrary to this, since the laser light of theLiDAR sensor has a strong directivity, the laser light tends to beradiated only to part of the screen. This LiDAR is a sensor whichdetects scattered light of laser radiation emitted in the form of pulsesto measure the distances of remote objects. Particularly, the LiDAR hasattracted attention as one of indispensable sensors required forincreasing precision of automatic driving. In what follows, the basicfeatures of the LiDAR are explained.

The LiDAR makes use of near-infrared micropulse light (for example,wavelength of 905 nm) as laser light. The LiDAR includes a scanner andan optical system which are constructed by, for example, a motor,mirrors and lenses. On the other hand, a light receiving unit and asignal processing unit receive reflected light and calculate a distanceby signal processing.

In this case, the LiDAR employs a LiDAR scan device 114 which is calledTOF system (Time of Flight). This LiDAR scan device 114 outputs laserlight as an irradiation pulse Plu1 from a light emitting element 114 bthrough an irradiation lens 114 c on the basis of the control by a laserdriver 114 a as illustrated in FIG. 7. This irradiation pulse Plu1 isreflected by a measurement object Ob1 and enters a light receiving lens114 d as a reflected pulse Plu2, and detected by a light receivingdevice 114 e. The detection result of this light receiving device 114 eis output from the LiDAR scan device 114 as an electrical signal througha signal light receiving circuit 114 f. Such a LiDAR scan device 114emits ultrashort pulses of a rising time of several nano seconds and alight peak power of several tens Watt to an object to be measured, andmeasures the time t required for the ultrashort pulses to reflect fromthe object to be measured and return to the light receiving unit. If thedistance to the object is L and the velocity of light is c, the distanceL is calculated by the following equation.

L=(c×t)/2

The basic operation of this LiDAR system is such that, as illustrated inFIGS. 8A to 8C, modulated laser light is emitted from the LiDAR scandevice 114, and reflected by a rotating mirror 114 g, distributed leftand right or rotating by 360° for scanning, and that the laser light asreflected by the object is returned and captured by the light receivingdevice 114 e of the LiDAR scan device 114 again. Finally, the capturedreflected light is used to obtain point group data PelY and PelXindicating signal levels corresponding to rotation angles. Incidentally,for example, the LiDAR system which is of a rotary type can emit laserlight by rotating a center unit as illustrated in FIG. 9 to performs360-degree scanning.

As described above, since the laser light of the LiDAR sensor has astrong directivity, even when laser light is radiated into the distance,the laser light tends to be radiated only to part of the screen.Accordingly, the shading unit 15 shown in FIG. 6 extracts, by the laserirradiated portion extraction unit 15 a, 3D profile only from a portionwhich is irradiated with laser light, performs shading of the extracted3D profile and outputs a shading image D106.

On the other hand, receiving 3D profile data D107 of the laserirradiated portion, likewise, the depth image generation unit 16 outputsa depth image D108 extracted only from a portion which is irradiatedwith laser light by the laser irradiated portion extraction unit 16 a.FIG. 10 illustrates an example in which laser interfering portions areextracted, beams of laser light are emitted through 360 degrees from aLiDAR which is mounted on the top of a moving vehicle at the center ofthe image. The example shown in the same figure includes a vehicledetected in the upper left side of the screen by the beam illuminationreflected on the vehicle and a walker detected in the upper right sideof the screen by the beam illumination reflected on the walker.

Accordingly, for example, the shading unit 15 has to generate an imagecorresponding to a 3D profile of the vehicle shown in FIG. 10 as aresult of shading by a 3DCG technique. Incidentally, while a RGB imageis internally generated in the case of the first embodiment (FIG. 1) asdescribed above, since the reflected light of laser light has no colorcomponent such as RGB, the shading image D106 is output from the shadingunit 15 directly as a gray scale image in the case of the presentembodiment. Next, while the depth image of the first embodiment coversthe entirety of the screen, the depth image generation unit 16 generatesthe depth image D108 of only the portion on which laser light isreflected.

By the process as described above, the depth image D108 and the shadingimage D106 as a gray scale image are transmitted as output images of thepresent embodiment. These two output images can be used for imagerecognition and recognition function learning.

Third Embodiment

Next, a deep learning recognition system of a virtual image inaccordance with a third embodiment of the present invention will beexplained. In the case of the present embodiment, it makes it possibleto supply various sensors with virtual environment images in anenvironment in which imaging is actually impossible by applying thevirtual image system with a near infrared sensor as described in thefirst embodiment and the virtual image system with a LiDAR sensor asdescribed in the second embodiment to an AI recognition technique suchas a deep learning recognition system commonly used for automaticdriving or the like.

(Configuration of a Deep Learning Recognition System of Virtual Images)

FIG. 11 is a view for schematically showing the configuration of a deeplearning recognition system in which is employed a back propagation typeneural network currently supposed to have best results. The deeplearning recognition system in accordance with the present embodiment ismainly constructed with a neural network calculation unit 17 and a backpropagation unit 18.

The neural network calculation unit 17 is provided with a neural networkconsisting of a number of layers, as illustrated in FIG. 12, to whichare input the gray scale image D104 and the depth image D105 as theoutput shown in FIG. 1. Then, non-linear calculation is performed on thebasis of coefficients (608, 610) which are set in the neural network inadvance to obtain final outputs 611.

On the other hand, the back propagation unit 18 receives calculationdata D110 which is a calculation result from the neural networkcalculation unit 17, and calculates error from teacher data which is thecomparison target (for example, an irradiation image, a depth image orthe like data on the basis of actual photography can be used). Thesystem as illustratively shown in FIG. 11 receives gray scale image D111as teacher data for the gray scale image D104, and receives depth imageD112 as teacher data for the depth image D105.

In this case, arithmetic operations are performed in accordance with theback propagation method in the back propagation unit 18. This backpropagation method calculates how much there is error between teacherdata and output data of the neural network, and has the result thereofpropagate backward again from the output side in the input direction. Inthe case of the present embodiment, receiving the error data D109 whichis fed back, the neural network calculation unit 17 performspredetermined calculation again, and inputs the result thereof to theback propagation unit 18. The above process in loop is repeated untilthe error data becomes smaller than a predetermined threshold, and theneural network calculation is finished when the error data has beenconverged fully.

When the above-mentioned process is completed, the coefficient values(608, 610) in the neural network in the neural network calculation unit17 are determined, and it is possible to perform deep learningrecognition for an actual image with this neural network.

Incidentally, while deep learning recognition in the case of the presentembodiment is illustratively described for the output image of the nearinfrared light image as described in the first embodiment, it ispossible to perform, completely in the same way, deep learningrecognition for the output image of a LiDAR sensor as described in thesecond embodiment by the similar technique. In such a case, the inputimages in the left side of FIG. 11 are the shading image D106 and thedepth image D108 shown in FIG. 6.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be explained. Inthe case of the second embodiment as described above, of the outputimages of the virtual image system utilizing a LiDAR sensor, the depthimage D108 is output from the depth image generation unit 16. As anevaluation point of this simulation system, it is very important howmuch accuracy this depth image has as a distance image actually obtainedwith assumed laser light. In the present embodiment, an example in whichthe present invention is applied to an evaluation system for evaluatingthis depth image will be explained.

(Configuration of a Depth Image Evaluation System)

As shown in FIG. 13, the depth image evaluation system in accordancewith the present embodiment is constructed as an evaluation means forevaluating the depth image D108 output from the depth image generationunit 16 as described above, and includes a TOF calculation unit 19, adistance image generation unit 20 and a comparison evaluation unit 21.

The TOF calculation unit 19 is a module for calculating TOF informationwhich includes TOF values and the like with respect to the depth imageD108 generated by the depth image generation unit 16. The TOF valuecorresponds to a delay time which is a time difference between emissionof a projection pulse from a light source and reception of theprojection pulse by a sensor as a light reception pulse after reflectionon the subject. This delay time is output from the TOF calculation unit19 as a TOF value D113.

The distance image generation unit 20 is a module for acquiring a TOF ofeach point of a laser irradiated portion on the basis of the TOF valuecalculated by the TOF calculation unit 19, calculating the distance L toeach point on the basis of the delay time of the each point, andgenerating a distance image D114 which represents the distance L to eachpoint by an image.

The comparison evaluation unit 21 is a module for performing comparisoncalculation between the distance image D114 generated by the distanceimage generation unit 20 and the depth image D108 as input from thedepth image generation unit 16, and performing evaluation on the basisof the result of comparison including the matching degree therebetween.The method of comparison can be performed by the use of absolute valuemean square error or the like which is generally used. The greater thevalue of the comparison result, the greater the difference therebetween,so that it is possible to evaluate how much the depth image based on 3DCG is close to the distance image generated by actually assuming TOF oflaser light.

(Operation of the Depth Image Evaluation System)

Next, the operation of the depth image evaluation system having thestructure as described above will be explained.

After receiving the depth image D108 generated by the depth imagegeneration unit 16, the TOF calculation unit 19 calculates the TOF. ThisTOF is “t” described with respect to FIG. 7. Specifically, after a lightsource emits laser light as a projection pulse as illustrated in FIG.14A, the projection pulse is reflected by the subject, and then receivedby the sensor as a light reception pulse. The time difference of thisprocess is measured. This time difference corresponds to the delay timebetween the projection pulse and the light reception pulse asillustrated in FIG. 14B.

As has been discussed above, the TOF value D113 calculated by the TOFcalculation unit 19 shown in FIG. 6 is output. Once the TOF of eachpoint of a laser irradiated portion is calculated by the TOF calculationunit 19, the distance L to each point can be obtained by backcalculation in accordance with the following equation.

L=(½)×c×t

(c: the velocity of light, t: TOF)

In accordance with the above equation, the distance image D114 of eachpoint of the irradiated image portion is generated by the distance imagegeneration unit 20. Thereafter, comparison calculation is performedbetween the depth image D108 and the distance image D114. The comparisonmeans can be implemented with absolute value mean square error or thelike which is generally used. The greater the value of the comparisonresult, the greater the difference therebetween, so that it is possibleto evaluate how much the depth image based on 3D CG is close to thedistance image generated by actually assuming TOF (this is correct) oflaser light.

A comparison result D115 may be output as a numeric value such as anabsolute value mean square error as described above or a signalindicative that both are not matched after the threshold process. In thelatter case, for example, the result may be fed back to the 3D modelingunit 11 shown in FIG. 6 followed by performing modeling again. Byrepeating this process until a predetermined approximation level isattained, a depth image can be generated on the basis of high precision3D CG.

Fifth Embodiment

Next, a fifth embodiment of the present invention will be explained.While each of the first to the fourth embodiment is related to the meansfor generating a near infrared ray or LiDAR sensor virtual image, thepresent embodiment is related to the explanation of control to actuallyperform automatic driving on a real time base by the use of thesevirtual images. In the case of the present embodiment, an example isdescribed in the case where the simulator system of the presentinvention is applied to the machine learning and test of an imagerecognition function module of an automated vehicle driving system.

In this description, the automated driving system is a system such asADAS (advanced driver assistance system) or the like to detect and avoidthe possibility of an accident in advance, and performs control todecrease the speed of the vehicle and avoid the objects and the like byrecognizing a camera image (real image) acquired with a camera actuallymounted on a vehicle to detect objects such as other vehicles, walkersand a traffic signal in accordance with an image recognition techniquefor the purpose of realizing automatic traveling of vehicles.

(Overall Configuration of Vehicle Synchronization Simulator System)

FIG. 15 is a schematic representation showing the overall configurationof the simulator system in accordance with the present embodiment. Thesimulator system in accordance with the present embodiment performssimulation programs with respect to a single or a plurality ofsimulation objects, and performs the machine learning and test of thesesimulation programs. As illustrated in FIG. 15, this simulator systemincludes a simulation server 2 located on a communication network 3, andconnected with an information processing terminal 1 a and avehicle-mounted device 1 b for generating or acquiring the position ofown vehicle through the communication network 3.

The communication network 3 is an IP network using the communicationprotocol TCP/IP, and a distributed communication network which isconstructed by connecting a variety of communication lines (a publicnetwork such as a telephone line, an ISDN line, an ADSL line or anoptical line, a dedicated communication line, the third generation (3G)communication system such as WCDMA (registered trademark) and CDMA2000,the fourth generation (4G) communication system such as LTE, the fifthgeneration (5G) or later communication system, and a wirelesscommunication network such as wifi (registered trademark) or Bluetooth(registered trademark)). This IP network includes a LAN such as a homenetwork, an intranet (a network within a company) based on 10BASE-T,100BASE-TX or the like. Alternatively, in many cases, simulator softwareis installed in the PC 1 a. In this case, simulation can be performed bysuch a PC alone.

The simulator server 2 is implemented with a single server device or agroup of server devices each of which has functions implemented by aserver computer or software capable of performing a variety ofinformation processes. This simulator server 2 includes a servercomputer which executes server application software, or an applicationserver in which is installed middleware for managing and assistingexecution of an application on such a computer.

Furthermore, the simulator server 2 includes a Web server whichprocesses a http response request from a client device. The Web serverperforms data processing and the like, and acts as an intermediary to adatabase core layer in which a relational database management system(RDBMS) is executed as a backend. The relational database server is aserver in which a database management system (DBMS) operates, and hasfunctions to transmit requested data to a client device and anapplication server (AP server) and rewrite or delete data in response toan operation request.

The information processing terminal 1 a and the vehicle-mounted device 1b are client devices connected to the communication network 3, andprovided with arithmetic processing units such as CPUs to provide avariety of functions by running a dedicated client program 5. Thisinformation processing terminal may be implemented with a generalpurpose computer such as a personal computer or a dedicated devicehaving necessary functions, and includes a smartphone, a mobilecomputer, PDA (Personal Digital Assistance), a cellular telephone, awearable terminal device, or the like.

This information processing terminal 1 a or the vehicle-mounted device 1b can access the simulator server 2 through the dedicated client program5 to transmit and receive data. Part or entirety of this client program5 is involved in a driving simulation system and a vehicle-mountedautomated driving system, and executed to recognize images captured by avehicle-mounted camera, or captured scenery images (including CG motionpictures in the case of the present embodiment) and the like by the useof an image recognition technique to detect objects such as othervehicles, walkers and a traffic signal in the images, calculate thepositional relationship between own vehicle and the object on the basisof the recognition result, and performs control to decrease the speed ofthe vehicle and avoid the objects and the like in accordance with thecalculation result. Incidentally, the client program 5 of the presentembodiment has the simulator server 2 perform an image recognitionfunction, and calculates or acquires the positional information of ownvehicle by having the own vehicle virtually travel on a map inaccordance with the recognition result of the simulator server 2 orhaving the own vehicle actually travel on the basis of the automaticdriving mechanism of a vehicle positional information calculation unit51 shown in FIGS. 18A and 18B to change the positional information ofthe own vehicle.

(Configuration of Each Device)

Next, the configuration of each device will specifically be explained.FIG. 16 is a block diagram for showing the internal structure of theclient device in accordance with the present embodiment. FIG. 17 is ablock diagram for showing the internal structure of the simulator serverin accordance with the present embodiment. Meanwhile, in the context ofthis document, the term “module” is intended to encompass any functionunit capable of performing necessary operation, as implemented withhardware such as a device or an apparatus, software capable ofperforming the functionality of the hardware, or any combinationthereof.

(1) Configuration of the Client Device

The information processing terminal 1 a can be implemented with ageneral purpose computer such as a personal computer or a dedicateddevice. On the other hand, the vehicle-mounted device 1 b may be ageneral purpose computer such as a personal computer, or a dedicateddevice (which can be regarded as a car navigation system) such as anautomated driving system. As illustrated in FIG. 16, specifically, theinformation processing terminal 1 a is provided with a CPU 102, a memory103, an input interface 104, a storage device 101, an output interface105 and a communication interface 106. Meanwhile, in the case of thepresent embodiment, these elements are connected to each other through aCPU bus to exchange data thereamong.

The memory 103 and the storage device 101 accumulate data on a recordingmedium, and read out accumulated data from the recording medium inresponse to an request from each device. The memory 103 and the storagedevice 101 may be implemented, for example, by a hard disk drive (HDD),a solid state drive (SSD), a memory card, and the like. The inputinterface 103 is a module for receiving operation signals from anoperation device such as a keyboard, a pointing device, a touch panel orbuttons. The received operation signals are transmitted to the CPU 102so that it is possible to perform operations of an OS or eachapplication. The output interface 105 is a module for transmitting imagesignals and sound signals to output an image and sound from an outputdevice such as a display or a speaker.

Particularly, in the case where the client device is a vehicle-mounteddevice 1 b, this input interface 104 is connected to a system such asthe above ADAS for automatic driving system, and also connected to animage sensor such as a camera 104 a or the like mounted on a vehicle, ora various sensor means such as a LiDAR sensor, a millimeter wave sensor,an infrared sensor or the like, for the purpose of realizing theautomated driving traveling of a vehicle.

The communication interface 106 is a module for transmitting andreceiving data to/from other communication devices on the basis of acommunication system including a public network such as a telephoneline, an ISDN line, an ADSL line or an optical line, a dedicatedcommunication line, the third generation (3G) communication system suchas WCDMA (registered trademark) and CDMA2000, the fourth generation (4G)communication system such as LTE, the fifth (5G) generation or latercommunication system, and a wireless communication network such as wifi(registered trademark) or Bluetooth (registered trademark)).

The CPU 102 is a device which performs a variety of arithmeticoperations required for controlling each element to virtually build avariety of modules on the CPU 102 by running a variety of programs. AnOS (Operating System) is executed and run on the CPU 102 to performmanagement and control of the basic functions of the informationprocessing terminals 1 a to 1 c, 4 and 5. Also, while a variety ofapplications can be executed on this OS, the basic functions of theinformation processing terminal are managed and controlled by runningthe OS program on the CPU 102, and a variety of function modules arevirtually built on the CPU 102 by running applications on the CPU 102.

In the case of the present embodiment, a client side execution unit 102a is formed by executing the client program 5 on the CPU 102 to generateor acquire the positional information of own vehicle on a virtual map ora real map, and transmit the positional information to the simulatorserver 2. The client side execution unit 102 a receives the recognitionresult of scenery images (including CG motion pictures in the case ofthe present embodiment) obtained by the simulator server 2, calculatethe positional relationship between own vehicle and the object on thebasis of the received recognition result, and performs control todecrease the speed of the vehicle and avoid the objects and the like onthe basis of the calculation result.

(2) Configuration of the Simulator Server

The simulator server 2 in accordance with the present embodiment is agroup of server devices which provide a vehicle synchronizationsimulator service through the communication network 3. The functions ofeach server device can be implemented by a server computer capable ofperforming a variety of information processes or software capable ofperforming the functions. Specifically, as illustrated in FIG. 17, thesimulator server 2 is provided with a communication interface 201, a UDPsynchronization control unit 202, a simulation execution unit 205, a UDPinformation transmitter receiver unit 206, and various databases 210 to213.

The communication interface 201 is a module for transmitting andreceiving data to/from other devices through the communication network 3on the basis of a communication system including a public network suchas a telephone line, an ISDN line, an ADSL line or an optical line, adedicated communication line, the third generation (3G) communicationsystem such as WCDMA (registered trademark) and CDMA2000, the fourthgeneration (4G) communication system such as LTE, the fifth (5G)generation or later communication system, and a wireless communicationnetwork such as wifi (registered trademark) or Bluetooth (registeredtrademark)).

As shown in FIGS. 18A and 18B, the UDP synchronization control unit 202is a module for controlling synchronization between a calculationprocess to calculate the positional information of own vehicle byvarying the position of the own vehicle in the client device 1 side, andan image generation process and an image recognition process in thesimulator server 2 side. The vehicle positional information calculationunit 51 of the client device 1 acquires the recognition result of animage recognition unit 204 through the UDP information transmitterreceiver unit 206, generates control signals for controlling vehiclebehavior by the use of the acquired recognition result, changes/modifiesthe positional information of own vehicle on the basis of the generatedcontrol signals.

The UDP information transmitter receiver unit 206 is a module fortransmitting and receiving data to/from the client side execution unit102 a of the client device 1 in cooperation. In the case of the presentembodiment, the positional information is calculated or acquired in theclient device 1 side, and packetized in a particular format. While thepacketized data is transmitted to the simulator server 2 through anetwork or a transmission bus in a particular device, the packet data isreceived and depacketized by the simulator server 2, and thedepacketized data is input to an image generation unit 203 to generateimages. Meanwhile, in the case of the present embodiment, the UDPinformation transmitter receiver unit 206 transmits and receives, by theuse of UDP (User Datagram Protocol), signals which are transmitted andreceived among the respective devices with the UDP synchronizationcontrol unit 202.

The above various databases include a map database 210, a vehicledatabase 211 and a drawing database 212. Incidentally, these databasescan be referred to each other by a relational database management system(RDBMS).

The simulation execution unit 205 is a module for generating asimulation image reproducing an area specified on the basis ofpositional information generated or acquired by the positionalinformation acquisition means of the client device 1 and transmitted tothe simulator server 2, and recognizing and detecting particular objectsin the generated simulation image by the use of the recognition functionmodule. Specifically, the simulation execution unit 205 is provided withthe image generation unit 203 and the image recognition unit 204.

The image generation unit 203 is a module for acquiring the positionalinformation acquired or calculated by the positional informationacquisition means of client device 1 and generating a simulation imagefor reproducing, by a computer graphics technique, an area (scenerybased on latitude and longitude coordinates of a map, and direction anda view angle) specified on the basis of the positional information. Thesimulation image generated by this image generation unit 203 istransmitted to the image recognition unit 204. Incidentally, this imagegeneration unit 203 can be implemented as the near infrared ray virtualimage generation system as explained in the first embodiment or theLiDAR virtual image generation system as explained in the secondembodiment, and the image recognition unit 204 may receive variousvirtual images generated by these systems in accordance with a computergraphics technique.

The image recognition unit 204 is a module for recognizing and detectingparticular objects in the simulation image generated by the imagegeneration unit 203 with the recognition function module 204 a which isunder test or machine learning. The recognition result information D06of this image recognition unit 204 is transmitted to the vehiclepositional information calculation unit 51 of the client device 1. Theimage recognition unit 204 is provided with a learning unit 204 b toperform machine learning of the recognition function module 204 a.

This recognition function module 204 a is a module for acquiring animage acquired with a camera device or CG generated by the imagegeneration unit 203, hierarchically extracting a plurality of featurepoints in the acquired image, and recognizing objects from thehierarchical combination patterns of the extracted feature points. Thelearning unit 204 b promotes diversification of extracted patterns andimproves learning efficiency by inputting images captured by the abovecamera device or virtual CG images to extract feature points of imageswhich are difficult to image and reproduce in practice.

This recognition function module 204 a of the image recognition unit maybe implemented by applying the neural network calculation unit 17 of thevirtual image deep learning recognition system as explained in the thirdembodiment, and the learning unit 204 b may be implemented by applyingthe back propagation unit 18 as described above.

(Method of the Vehicle Synchronization Simulator System)

The vehicle synchronization simulation method can be implemented byoperating the vehicle synchronization simulator system having thestructure as described above. FIGS. 18A and 18B are block diagrams forshowing the configuration and operation of image generation and imagerecognition in accordance with the present embodiment. FIG. 19 is a flowchart for showing the procedure of a synchronization simulator inaccordance with the present embodiment.

At first, the vehicle positional information calculation unit 51acquires vehicle positional information D02 of own vehicle (S101).Specifically, the client program 5 is executed in the client device 1side to input a various data group D01 such as map information andvehicle initial data to the vehicle positional information calculationunit 51. Next, the positional information of own vehicle on a virtualmap or an actual map is calculated (generated) or acquired by the use ofthe data group D01. The result is transmitted to the simulationexecution unit 205 of the simulator server 2 (S102) as vehiclepositional information D02 through the UDP synchronization control unit202 or the UDP information transmitter receiver unit 206.

Specifically speaking, the vehicle positional information calculationunit 51 transmits the vehicle positional information D02 of own vehicleto the UDP synchronization control unit 202 in accordance with thetiming of a control signal D03 from the UDP synchronization control unit202. Of initial data of the vehicle positional information calculationunit 51, map data, the positional information of own vehicle in the map,the rotation angle and diameter of a wheel of the vehicle body frame andthe like information, can be loaded from the predetermined storagedevice 101. The UDP synchronization control unit 202 and the UDPinformation transmitter receiver unit 206 transmit and receive datafrom/to the client side execution unit 102 a of the client device 1 incooperation. Specifically, the UDP synchronization control unit 202 andthe UDP information transmitter receiver unit 206 transmit the vehiclepositional information D02 calculated or acquired in the client device 1side to the simulator server 2 as packet information D04 packetized in aparticular format with a various data group including vehicleinformation.

While this packetized data is transmitted through a network or atransmission bus in a particular device, the packet data is received anddepacketized by the simulator server 2 (S103), and the depacketized dataD05 is input to the image generation unit 203 of the simulationexecution unit 205 to generate CG images. In this case, the UDPinformation transmitter receiver unit 206 transmits and receives thepacketized packet information D04 of a various data group includingvehicle information among the respective devices by the UDPsynchronization control unit 202 according to UDP (User DatagramProtocol).

Specifically describing, the UDP synchronization control unit 202converts the various data group into the packetized packet informationD04 by UDP packetizing the vehicle positional information D02 of ownvehicle. Thereby, data transmission and reception by the use of the UDPprotocol becomes easy. At this time, UDP (User Datagram Protocol) willbe described to some extent. Generally speaking, while TCP is highreliable and connection oriented and performs windowing control,retransmission control and congestion control, UDP is a connection-lessprotocol which has no mechanism to secure reliability but has asubstantial advantage due to low delay because the process is simple. Inthe case of the present embodiment, since low delay is required duringtransmitting data among the constituent elements, UDP is employedinstead of TCP. Alternatively, RTP (Realtime Transport Protocol) may beused as the most common protocol for voice communication and videocommunication at the present time.

Next, the vehicle positional information D02 of own vehicle specificallycontains, for example, the following information.

-   -   Positional information (three dimensional coordinates (X, Y, Z)        of road surface absolute position coordinates or the like) of        own vehicle    -   Euler angles of own vehicle    -   Positional information (three dimensional coordinates (X, Y, Z)        of tires of road surface absolute position coordinates or the        like) of tires    -   Wheel rotation angle    -   Stamping margin of a brake and a steering wheel

Receiving the vehicle positional information D02, the UDP informationtransmitter receiver unit 206 transmits data D05 necessary mainly forgenerating a vehicle CG image, from among information about the vehicle,e.g., XYZ coordinates as the positional information of the vehicle, XYZcoordinates as the positional information of tires, Euler angles andother various information.

Then, the packet information D04 as UDP packets of the various datagroup is divided into a packet header and a payload of a data body by adepacketizing process in the UDP information transmitter receiver unit206. In this case, the UDP packet data can be exchanged by transmissionbetween places remote from each other through a network or transmissioninside a single apparatus such as a simulator through a transmissionbus. The data D05 corresponding to a payload is input to the imagegeneration unit 203 of the simulation execution unit 205 (S104).

In the simulation execution unit 205, the image generation unit 203acquires positional information acquired or calculated by the positionalinformation acquisition means of the client device 1 as the data D05,and generates a simulation image for reproducing, by a computer graphicstechnique, an area (scenery based on latitude and longitude coordinatesof a map, a direction and a view angle) specified on the basis of thepositional information (S105). The image D13 for simulation generated bythis image generation unit 203 is transmitted to the image recognitionunit 204.

The image generation unit 203 generates a realistic image by apredetermined image generation method, for example, a CG imagegeneration technique which makes use of the latest physically basedrendering (PBR) technique. The recognition result information D06 isinput to the vehicle positional information calculation unit 51 againand used, e.g., for calculating the positional information of ownvehicle for determining the next behavior of the own vehicle.

The image generation unit 203 generates objects such as a road surface,buildings, a traffic signal, other vehicles and walkers by, for example,a CG technique making use of the PBR technique. This can be understoodas feasible with the latest CG technique from the fact that objects suchas described above are reproduced in a highly realistic manner in atitle of a game machine such as PlayStation. In many cases, objectimages other than own vehicle are stored already as initial data.Particularly, in an automatic driving simulator, a large amount ofsample data such as a number of highways and general roads is stored ina database which can readily be used.

Next, the image recognition unit 204 recognizes and extracts particulartargets, as objects, by the use of the recognition function module 204 awhich is under test or machine learning from simulation images generatedby the image generation unit 203 (S106). In this case, if there is noobject which is recognized (“N” in step S107), the process proceeds tothe next time frame (S109), and the above processes S101 to S107 arerepeated (“Y” in step S109) until all the time frames are processed (“N”in step S109).

On the other hand, if there is an object which is recognized (“Y” instep S107), the recognition result of this image recognition unit 204 istransmitted to the vehicle positional information calculation unit 51 ofthe client device 1 as the recognition result information D06. Thevehicle positional information calculation unit 51 of the client device1 acquires the recognition result information D06 of the imagerecognition unit 204 through the UDP information transmitter receiverunit 206, generates control signals for controlling vehicle behavior bythe use of the acquired recognition result, changes/modifies thepositional information of own vehicle on the basis of the generatedcontrol signals (S108).

Specifically describing, the simulation image D13 which is generatedhere is input to the image recognition unit 204 and, as alreadydescribed above, objects are recognized and detected by, for example, arecognition technique such as deep learning. The recognition results asobtained are given as area information in a screen (for example,two-dimensional XY coordinates of an extracted rectangular area) such asother vehicles, walkers, road markings and a traffic signal.

When running a simulator for automatic driving, there are a number ofobjects such as other vehicles, walkers, buildings and a road surface ina screen in which an actual vehicle is moving. Automatic driving isrealized, for example, by automatically turning the steering wheel,stepping on the accelerator, applying the brake and so on whileobtaining realtime information obtained from a camera mounted on thevehicle, a millimeter wave sensor, a radar and other sensors.

Accordingly, in the case of the near infrared light image described inthe embodiment 1, a recognition technique such as deep learning asdescribed in the embodiment 3 is used to recognize and discriminateobjects necessary for automatic driving such as other vehicles, walkers,road markings and a traffic signal from among objects displayed on ascreen.

For example, when another vehicle cuts in front of own vehicle, theimage recognition unit 204 detects the approach by an image recognitiontechnique, and outputs the recognition result information D06 of therecognition result to the vehicle positional information calculationunit 51. The vehicle positional information calculation unit 51 changesthe positional information of own vehicle by turning the steering wheelto avoid the another vehicle, applying the brake to decelerate ownvehicle or performing the like operation. In an another case where awalker suddenly runs out in front of own vehicle, likewise, the vehiclepositional information calculation unit 51 changes the positionalinformation of own vehicle by turning the steering wheel to avoid thiswalker, applying the brake to decelerate own vehicle or performing thelike operation.

Meanwhile, in the above described configuration, it is assumed that datais transmitted in a cycle of 25 msec (25 msec is only one example)according to the UDP protocol from the vehicle positional informationcalculation unit 51 to the simulation execution unit 205 through the UDPsynchronization control unit 202 and the UDP information transmitterreceiver unit 206.

Incidentally, the need of “synchronizing model” which is acharacteristic feature of the present invention exists because thevehicle positional information of the next time frame is determined onthe basis of the output result from the simulation execution unit 205 sothat the behavior of a real vehicle cannot be simulated unless theentirety can be synchronously controlled. In the above example,transmission is performed in a cycle of 25 msec. However, ideal delay iszero which is practically impossible, so that UDP is employed to reducethe delay time associated with transmission and reception.

Generally speaking, in the case of an automatic driving simulator, testhas to be conducted with a very large amount of motion image frames. Itis an object of the present embodiment to substitute CG images nearer toactual photographs for an unquestioning amount which cannot be coveredby real driving. Accordingly, it is necessary to guarantee operations inresponse to a long sequence of video sample data.

In the case of the present embodiment, the learning unit 204 bdiversifies extracted pattern to improve learning efficiency byinputting, in addition to images taken by a vehicle mounted cameraduring real driving, virtual CG images generated by the image generationunit 203 to the recognition function module 204 a to extract the featurepoints of images which are difficult to take and reproduce. Therecognition function module 204 a acquires images taken by the cameradevice and CG images, hierarchically extracts a plurality of featurepoints in the acquired images, and recognizes objects by the deeplearning recognition technique already described in the embodiment 3 onthe basis of combinational hierarchic patterns of the extracted objects.

Sixth Embodiment

In what follows, with reference to the accompanying drawings, a sixthembodiment of the system in accordance with the present invention willbe explained in detail. FIG. 20 is a schematic representation showingthe overall configuration of the system in accordance with the presentembodiment. FIG. 21 is a block diagram for showing the internalstructure of the device in accordance with the present embodiment. Whilethe fifth embodiment as described above is an embodiment in which ownvehicle is limited to a single vehicle, the present embodiment isdirected to an example in which the positional information of number ofvehicles are simultaneously processed in parallel.

As shown in FIG. 20, in the case of the present embodiment, a pluralityof client devices 1 c to 1 f are connected to the simulator server 2,and as shown in FIG. 21, while the UDP synchronization control unit 202and the UDP information transmitter receiver unit 206 serve as commonelements in the simulator server 2, in correspondence with the number ofvehicles to be simulated, there are vehicle positional informationcalculation units 51 c to 51 f provided in the client devices 1 c to 1 frespectively and simulation execution units 205 c to 205 f provided inthe simulator server 2.

The vehicle positional information calculation units 51 c to 51 ftransmit vehicle positional information D02 c to D02 f to the UDPsynchronization control unit 202 with the timing of control signals D03c to D03 f. Next, the UDP synchronization control unit 202 converts thevehicle positional information D02 c to D02 f to packet information D04by UDP packetization. Thereby, data transmission and reception by theuse of the UDP protocol becomes easy. The packet information D04 isdivided into a packet header and a payload of a data body by adepacketizing process in the UDP information transmitter receiver unit206. In this case, the UDP packet data can be exchanged by transmissionbetween places remote from each other through a network or transmissioninside a single apparatus such as a simulator through a transmissionbus. The data D05 c to D05 f corresponding to a payload is input to thesimulation execution units 205 c to 205 f.

As has already been discussed above in the first embodiment, thesimulation execution units 205 c to 205 f generates a realistic image bya predetermined image generation method, for example, a CG imagegeneration technique which makes use of the latest physically basedrendering (PBR) technique. The recognition result information D06 c toD06 f is fed back to the vehicle positional information calculationunits 51 c to 51 f to change the position of each vehicle.

Incidentally, while there are four vehicle positional informationcalculation units 51 c to 51 f in the above example, this number is notlimited to four. However, if the number of vehicles to be supportedincreases, synchronization control as a result becomes complicated, andthere is a problem that when there occurs a substantial delay in acertain vehicle, the total delay time increases since the delay times ofthe vehicles are summed up. Accordingly, the configuration can bedesigned in accordance with the hardware scale, processing amount andother conditions of the simulator server.

Incidentally, while PC terminals 1 c to 1 f are remotely connected to avehicle synchronization simulator program 4 through the communicationnetwork 3 in FIG. 20, the PC terminals 1 c to 1 f can be operated in astand-alone manner by installing a program in a local recording mediumsuch as an HDD or an SDD. In this case, there are advantages in thattest can be performed with a low delay and that no influence ofcongestion troubles or the like need not be considered when a shortageof network band is caused.

Furthermore, while 1 c to 1 f are not limited to PC terminals, forexample, when test is conducted with actually moving vehicles, 1 c to 1f can be considered to refer to car navigation systems mounted on thetest vehicles. In this case, rather than recognizing the simulationimage D13 which is a CG image generated by the image generation unit 203of FIG. 18B, the learning unit 204 receives a live-action video in placeof the simulation image D13 so that the system can be used forevaluating the performance of the image recognition unit 204. This isbecause, while a human being can immediately and accurately recognize awalker and a vehicle in a live-action video, it is possible to verifywhether or not the image recognition unit 204 can output the same resultof extraction and recognition.

Seventh Embodiment

Furthermore, a seventh embodiment of the system in accordance with thepresent invention will be explained. In the case of the presentembodiment, another embodiment implemented with a plurality of sensorswill be explained with reference to FIG. 22. This FIG. 22 shows anexample in which different devices of sensors are installed. In the samefigure, it is assumed that one of deep learning recognition units isprovided for example for an image sensor such as a camera, and thatanother deep learning recognition unit is provided for example for anear infrared sensor or a LiDAR (Light Detection and Ranging).

As illustrated in FIG. 22, the first deep learning recognition unit 61is implemented with an image sensor unit, and the 3D graphicssynthesized image is a two-dimensional image. Accordingly, the deeplearning recognition means is provided with a function to recognize atwo-dimensional image. On the other hand, the next deep learningrecognition unit 62 makes use of 3D point group data obtained by a LiDARsensor. This 3D point group data is converted to a 3D graphic image inthe image generation unit 203.

The 3D point group data converted to the 3D graphic image as describedabove is point group data which is obtained by emitting laser light toall directions of 360 degrees from a LiDAR installed on the runningcenter vehicle shown in FIG. 10 and measuring the reflected light. Theintensity of color indicates the intensity of the reflected light.Accordingly, the area such as a gap in which no substance exists iscolored black because there is no reflected light.

Target objects such as an opposite running vehicle, a walker and abicycle can be acquired from actual point group data asthree-dimensional coordinate data, and therefore it is possible toeasily generate 3D graphic images of these target objects. Specifically,a plurality of polygon data items are generated by consistentlyprocessing point group data, and 3D graphics can be drawn by renderingthese polygon data items.

Then, the 3D point group data graphic image as generated by the abovemeans is input to the deep learning recognition unit 62, and recognizedby recognition means which has performed learning for 3D point groupdata in the deep learning recognition unit 62. Accordingly, differentmeans is used than the deep learning recognition means which hasperformed learning with images for image sensors as described above, andthis is substantially effective. This is because while it is likely thata vehicle which is very far away cannot be acquired by the image sensor,the LiDAR can acquire the size and profile of an oncoming vehicle evenat the front of several hundred meters. Conversely, while the LiDARmakes use of reflected light so that there is a problem that the LiDARis not effective for detecting a target object which is not reflective,there is not such a problem in the case of the image sensor.

As has been discussed above, there are provided a plurality of sensorshaving different characteristics or different device properties, and thelearning result synchronization unit 84 analyzes the recognition resultsthereof, and output the final recognition result D62. Incidentally, thissynchronization unit may be arranged outside, for example, in a networkcloud. This is because, while the number of sensors per one vehicledramatically increases in the future, and the computational load of thedeep learning recognition process increases, it is effective to performprocesses, which can be handled outside through a network, by a cloudhaving a large scale computing power, and feed back the results.

Incidentally, while virtual CG images are generated in the case of theembodiment shown in FIG. 22, as has been discussed in the firstembodiment, it is possible to perform deep learning recognition byinstalling this application system in an actual vehicle (like carnavigation system) and inputting information to the system fromdifferent types of sensors while actually imaging and driving thevehicle. FIG. 23 is a block diagram for showing an actual case of such asystem.

It is assumed that the object imaging devices are a LiDAR sensor and amillimeter wave sensor as described above besides the image sensorinstalled in a vehicle mounted camera. In the case of the image sensor,a high quality CG image is generated by a PBR technique as described inthe first embodiment with reference to parameters such as lightinformation extracted from a photographed image as acquired, and the CGimage is output from the image generation unit 203. On the other hand,in the case of the LiDAR sensor, a three-dimensional point group data isgenerated from the reflected light of laser light which is a beamemitted from the LiDAR sensor actually mounted on a vehicle. Then, animage as a 3D CG converted from this three-dimensional point group datais output from the above image generation unit 203.

In this way, CG images corresponding to a plurality of types of sensorsare emitted from the image generation unit 203, and the recognitionprocess thereof is performed in each deep learning recognition unit ofFIG. 23 by predetermined means. Also, while the above embodiment hasbeen explained with a LiDAR sensor as an example, it is also effectiveto make use of an infrared sensor as explained in the second embodiment.

EXPLANATION OF SYMBOLS

-   -   D01 . . . data group    -   D02 (D02 c-f) . . . vehicle positional information    -   D03 (D03 c-f) . . . control signal    -   D04 . . . packet information    -   D05 (D05 c-f) . . . data    -   D06 (D06 c-f) . . . recognition result information    -   D100 . . . scenario information    -   D101 . . . model data    -   D102 . . . modeling information    -   D103, D106 . . . shading image    -   D104 . . . gray scale image    -   D105, D108, D112 . . . depth image    -   D107 . . . 3D profile data    -   D109 . . . error data    -   D110 . . . calculation data    -   D111 . . . gray scale image    -   D113 . . . TOF value    -   D114 . . . distance image    -   D115 . . . comparison result    -   D13 . . . simulation image    -   D61 . . . 3D point group data graphic image    -   D62 . . . recognition result    -   1 . . . client device    -   1 a . . . information processing terminal    -   1 b . . . vehicle-mounted device    -   1 c-1 f . . . client device    -   2 . . . simulator servers    -   3 . . . communication networks    -   4 . . . vehicles synchronization simulator program    -   5 . . . client program    -   10 . . . scenario creation unit    -   114 . . . LiDAR scan device    -   114 a . . . laser driver    -   114 b . . . light emitting element    -   114 c . . . irradiation lens    -   114 d . . . light receiving lens    -   114 e . . . light receiving device    -   114 f . . . signal light receiving circuit    -   114 g . . . mirror    -   11 . . . 3D modeling unit    -   12 . . . 3D shading unit    -   13 . . . R image gray scale conversion unit    -   14 . . . depth image generation unit    -   15 . . . shading unit    -   15 a . . . laser irradiated portion extraction unit    -   16 . . . depth image generation unit    -   16 a . . . laser irradiated portion extraction unit    -   17 . . . neural network calculation unit    -   18 . . . back propagation unit    -   19 . . . TOF calculation unit    -   20 . . . distance image unit    -   21 . . . comparison evaluation unit    -   51 (51 c-f) . . . vehicle positional information calculation        unit    -   61-6 n . . . deep learning recognition unit    -   84 . . . learning result synchronization unit    -   101 . . . storage device    -   102 . . . CPU    -   102 a . . . client side execution unit    -   103 . . . memory    -   104 . . . input interface    -   105 . . . output interface    -   106, 201 . . . communication interface    -   202 . . . UDP synchronization control unit    -   203 . . . image generation unit    -   204 . . . image recognition unit    -   204 a . . . recognition function module    -   204 b . . . learning unit    -   205 . . . simulation execution unit    -   205 c-f . . . simulation execution unit    -   206 . . . UDP information transmitter receiver unit    -   210 . . . map database    -   210-213 . . . database    -   211 . . . vehicle database    -   212 . . . drawing database    -   402 . . . CPU    -   611 . . . output

What is claimed is:
 1. An image generation system of generating, ascomputer graphics, a virtual image which is input to a sensor unit,comprising: a scenario creation unit which creates a scenario relatingto locations and behaviors of objects existing in the virtual image; a3D modeling unit which performs modeling of each of the objects on thebasis of the scenario; a 3D shading unit which performs shading of eachmodel generated by the modeling unit and generates a shading image ofeach model; a component extraction unit which extracts and outputs apredetermined component contained in the shading image as a componentimage; and a depth image generation unit which generates a depth imagein which a depth is defined on the basis of three-dimensional profileinformation about each object in the component image.
 2. The imagegeneration system of claim 1 wherein the component is an R component ofan RGB image.
 3. The image generation system of claim 1 furthercomprising: a gray scale conversion unit which performs gray scaleconversion of the component.
 4. An image generation system ofgenerating, as computer graphics, a virtual image which is input to asensor unit, comprising: a scenario creation unit which creates ascenario relating to locations and behaviors of objects existing in thevirtual image; a 3D modeling unit which performs modeling of each of theobjects on the basis of the scenario; a 3D shading unit which performsshading of each model generated by the modeling unit and generates ashading image of each model; and a depth image generation unit whichgenerates a depth image in which a depth is defined on the basis ofthree-dimensional profile information about each of the objects, whereinthe shading unit is provided with: a function to perform shading only ofa predetermined portion of the model on which is reflected a light beamemitted from the sensor unit; and a function to output only athree-dimensional profile of the predetermined portion, and wherein thedepth image generation unit generates a depth image of each of theobjects on the basis of the three-dimensional profile of thepredetermined portion.
 5. The image generation system of claim 1 whereinthe sensor unit is a near infrared sensor.
 6. The image generationsystem of claim 1 wherein the sensor unit is a LiDAR sensor whichdetects reflected light of emitted laser light.
 7. The image generationsystem of claim 1 wherein the scenario creation unit is provided with amechanism to determine three-dimensional profile information of objects,behavior information of objects, material information of objects,parameter information of light sources, positional information ofcameras and positional information of sensors.
 8. The image generationsystem of claim 1 further comprising: a deep learning recognitionlearning unit which acquires, as teacher data, and performs training ofa neural network by back propagation on the basis of the componentimage, the depth image generated by the depth image generation unit, andthe teacher data.
 9. The image generation system of claim 4 furthercomprising: a deep learning recognition learning unit which acquires, asteacher data, an irradiation image and a depth image on the basis ofactual photography, and performs training of a neural network by backpropagation on the basis of the image obtained by the shading unit as aresult of shading, the depth image generated by the depth imagegeneration unit, and the teacher data.
 10. The image generation systemof claim 1 further comprising: a TOF calculation unit which calculates,as TOF information, a time required from irradiation of a light beam toreception of a reflected light thereof on the basis of the depth imagegenerated by the depth image generation unit; a distance imagegeneration unit which generates a distance image on the basis of the TOFinformation calculated by the TOF calculation unit; and a comparisonevaluation unit which compares the distance image generated by thedistance image generation unit and the depth image generated by thedepth image generation unit.
 11. The image generation system of claim 10wherein the modeling unit has a function to acquire the result ofcomparison by the comparison evaluation unit as feedback information,adjust conditions of the modeling on the basis of the acquired feedbackinformation, and perform modeling again.
 12. The image generation systemof claim 11 wherein the modeling unit repeats the modeling untilmatching error of the comparison result by the comparison evaluationunit becomes smaller than a predetermined threshold by repeatingacquisition of the feedback information on the basis of the modeling andthe comparison.
 13. A simulation system of a recognition function modulefor an image varying in correspondence with position shiftinginformation of a vehicle, comprising: a positional informationacquisition unit which acquires positional information of the vehicle inrelation to a surrounding object on the basis of a detection result by asensor unit; an image generation unit which generates a simulation imagefor reproducing an area specified by the positional information on thebasis of the positional information acquired by the positionalinformation acquisition unit; an image recognition unit which recognizesand detects a particular object by the recognition function module inthe simulation image generated by the image generation unit; apositional information calculation unit which generates a control signalfor controlling behavior of the vehicle by the use of the recognitionresult of the image recognition unit, and changes/modifies thepositional information of own vehicle on the basis of the generatedcontrol signal; and a synchronization control unit which controlssynchronization among the positional information acquisition unit, theimage generation unit, the image recognition unit and the positionalinformation calculation unit, wherein as the above vehicle, a pluralityof vehicles are set up for each of which the recognition functionoperates, wherein the positional information calculation unitchanges/modifies the positional information of each of the plurality ofvehicles by the use of information about the recognition result of therecognition unit, and wherein the synchronization control unit controlssynchronization among the positional information acquisition unit, theimage generation unit, the image recognition unit and the positionalinformation calculation unit for each of the plurality of vehicles. 14.A simulation system of a recognition function module for an imagevarying in correspondence with position shifting information of avehicle, comprising: a positional information acquisition unit whichacquires positional information of the vehicle in relation to asurrounding object on the basis of a detection result by a sensor unit;an image generation unit which generates a simulation image forreproducing an area specified by the positional information on the basisof the positional information acquired by the positional informationacquisition unit; an image recognition unit which recognizes and detectsa particular object by the recognition function module in the simulationimage generated by the image generation unit; a positional informationcalculation unit which generates a control signal for controllingbehavior of the vehicle by the use of the recognition result of theimage recognition unit, and changes/modifies the positional informationof own vehicle on the basis of the generated control signal; and asynchronization control unit which controls synchronization among thepositional information acquisition unit, the image generation unit, theimage recognition unit and the positional information calculation unit,wherein the simulation system is provided with a unit of generatingimages corresponding to a plurality of sensors, a recognition unitsupporting the generated images, a unit of performing thesynchronization control by the use of the plurality of the recognitionresults.
 15. A simulation system of a recognition function module for animage varying in correspondence with position shifting information of avehicle, comprising: a positional information acquisition unit whichacquires positional information of the vehicle in relation to asurrounding object on the basis of a detection result by a sensor unit;an image generation unit which generates a simulation image forreproducing an area specified by the positional information on the basisof the positional information acquired by the positional informationacquisition unit; an image recognition unit which recognizes and detectsa particular object by the recognition function module in the simulationimage generated by the image generation unit; a positional informationcalculation unit which generates a control signal for controllingbehavior of the vehicle by the use of the recognition result of theimage recognition unit, and changes/modifies the positional informationof own vehicle on the basis of the generated control signal; and asynchronization control unit which controls synchronization among thepositional information acquisition unit, the image generation unit, theimage recognition unit and the positional information calculation unit,wherein the simulation system provided with, as the image generationunit, the image generation system as recited in claim 1 or claim 4, andthe depth image generated by the depth image generation unit of theimage generation system is input to the image recognition unit as thesimulation image.
 16. The simulation system of claim 13 wherein thesynchronization control unit comprises: a unit of packetizing thepositional information in a particular format and transmitting thepacketized positional information; a unit of transmitting the packetizeddata through a network or a transmission bus in a particular device; aunit of receiving and depacketizing the packetized data; and a unit ofreceiving the depacketized data and generating an image.
 17. Thesimulation system of claim 13 wherein the synchronization control unittransmits and receives signals among the respective units in accordancewith UDP (User Datagram Protocol).
 18. The simulation system of claim 13wherein the positional information of the vehicle includes informationabout any of XYZ coordinates of road surface absolute positioncoordinates of the vehicle, XYZ coordinates of road surface absoluteposition coordinates of tires, Euler angles of own vehicle and a wheelrotation angle.
 19. The simulation system of claim 13 wherein the imagegeneration unit is provided with a unit of synthesizing athree-dimensional profile of the vehicle by computer graphics.
 20. Thesimulation system of claim 13 wherein the image generation unit isprovided with a unit of generating a different image for each sensorunit.
 21. The simulation system of claim 13 wherein there is provided,as the sensor unit, with any or all of an image sensor, a LiDAR sensor,a millimeter wave sensor and an infrared sensor.
 22. An image generationprogram for generating a virtual image to be input to a sensor unit ascomputer graphics, and causing a computer to function as: a scenariocreation unit which creates a scenario relating to locations andbehaviors of objects existing in the virtual image; a 3D modeling unitwhich performs modeling of each of the objects on the basis of thescenario; a 3D shading unit which performs shading of each modelgenerated by the modeling unit and generates a shading image of eachmodel; a component extraction unit which extracts and outputs apredetermined component contained in the shading image as a componentimage; and a depth image generation unit which generates a depth imagein which a depth is defined on the basis of three-dimensional profileinformation about each object in the component image.
 23. An imagegeneration program for generating a virtual image to be input to asensor unit as computer graphics, and causing a computer to function as:a scenario creation unit which creates a scenario relating to locationsand behaviors of objects existing in the virtual image; a 3D modelingunit which performs modeling of each of the objects on the basis of thescenario; a 3D shading unit which performs shading of each modelgenerated by the modeling unit and generates a shading image of eachmodel; a depth image generation unit which generates a depth image inwhich a depth is defined on the basis of three-dimensional profileinformation about each object, wherein the shading unit is providedwith: a function to perform shading only of a predetermined portion ofthe model on which is reflected a light beam emitted from the sensorunit; and a function to output only a three-dimensional profile of thepredetermined portion, wherein the depth image generation unit generatesa depth image of each of the objects on the basis of thethree-dimensional profile of the predetermined portion.
 24. A simulationprogram of a recognition function module for an image varying incorrespondence with position shifting information of a vehicle, causinga computer to function as: a positional information acquisition unitwhich acquires positional information of the vehicle; an imagegeneration unit which generates a simulation image for reproducing anarea specified by the positional information on the basis of thepositional information acquired by the positional informationacquisition unit; an image recognition unit which recognizes and detectsa particular object by the recognition function module in the simulationimage generated by the image generation unit; a positional informationcalculation unit which generates a control signal for controllingbehavior of the vehicle by the use of the recognition result of theimage recognition unit, and changes/modifies the positional informationof own vehicle on the basis of the generated control signal; and asynchronization control unit which controls synchronization among thepositional information acquisition unit, the image generation unit, theimage recognition unit and the positional information calculation unit,wherein the simulation program provided with, as the image generationunit, the image generation program as recited in claim 22 or claim 23,and the depth image generated by the depth image generation unit of theimage generation system is input to the image recognition unit as thesimulation image.
 25. An image generation method of generating, ascomputer graphics, a virtual image which is input to a sensor unit,comprising: a scenario creation step of creating a scenario relating tolocations and behaviors of objects existing in the virtual image by ascenario creation unit; a 3D modeling step of performing modeling ofeach of the objects on the basis of the scenario by a 3D modeling unit;a 3D shading step of performing shading of each model generated in the3D modeling step and generating a shading image of each model by a 3Dshading unit; a component extraction step of extracting and outputting apredetermined component contained in the shading image as a componentimage by a component extraction unit; and a depth image generation stepof generating, by a depth image generation unit, a depth image in whicha depth is defined on the basis of three-dimensional profile informationabout each object in the component image.
 26. An image generation methodof generating, as computer graphics, a virtual image which is input to asensor unit, comprising: a scenario creation step of creating a scenariorelating to locations and behaviors of objects existing in the virtualimage by a scenario creation unit; a 3D modeling step of performingmodeling of each of the objects on the basis of the scenario by a 3Dmodeling unit; a 3D shading step of performing shading of each modelgenerated in the 3D modeling step and generating a shading image of eachmodel by a 3D shading unit; a depth image generation step of generating,by a depth image generation unit, a depth image in which a depth isdefined on the basis of three-dimensional profile information about eachobject in the shading image, wherein the shading unit is provided with:a function to perform shading only of a predetermined portion of themodel on which is reflected a light beam emitted from the sensor unit;and a function to output only a three-dimensional profile of thepredetermined portion, wherein the depth image generation unit generatesa depth image of each of the objects on the basis of thethree-dimensional profile of the predetermined portion.
 27. Thesimulation method includes, as the image generation step, the imagegeneration method as recited in claim 25, wherein the depth imagegenerated by the depth image generation unit in the image generationmethod is input to the image recognition unit as the simulation image.28. A simulation method of a recognition function module for an imagevarying in correspondence with position shifting information of avehicle, comprising: a positional information acquisition step ofacquiring positional information of the vehicle by a positionalinformation acquisition unit; an image generation step of generating, byan image generation unit, a simulation image for reproducing an areaspecified by the positional information on the basis of the positionalinformation acquired in the positional information acquisition step; animage recognition step of recognizing and detecting a particular objectby the recognition function module in an image recognition unit in thesimulation image generated in the image generation step; a positionalinformation calculation step of generating a control signal forcontrolling behavior of the vehicle by the use of the recognition resultin the image recognition step and changing/modifying the positionalinformation of own vehicle on the basis of the generated control signal,by a positional information calculation unit; and a synchronizationcontrol step of controlling synchronization among the positionalinformation acquisition unit, the image generation unit, the imagerecognition unit and the positional information calculation unit by asynchronization control unit, wherein the simulation method includes, asthe image generation step, the image generation method as recited inclaim 25 or claim 26, and the depth image generated by the depth imagegeneration unit in the image generation method is input to the imagerecognition unit as the simulation image.